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Highlights 


United  States  General  Accounting  Office  August  2002 

CUSTOMS  SERVICE  MODERNIZATION 
Third  Expenditure  Plan  Meets  Legislative 
Conditions,  but  Cost  Estimating 
Improvements  Needed 

Highlights  of  GAO-02-908,  a  report  to  the  Subcommittee  on  Treasury  and  General  Government,  Senate 
Committee  on  Appropriations,  and  the  Subcommittee  on  Treasury,  Postal  Service,  and  General 
Government,  House  Committee  on  Appropriations. 


Why  GAO  Did  This  Study 

The  U.S.  Customs  Service  has 
begun  a  multiyear,  multibillion- 
dollar  project:  the  Automated 
Commercial  Environment  (ACE), 
a  new  import  processing  system 
that  is  planned  to  support 
effective  and  efficient  movement 
of  goods  into  the  United  States. 
By  congressional  mandate, 
Customs’  expenditure  plans  for 
ACE  must  meet  certain 
conditions,  including  being 
reviewed  by  GAO.  This  study 
addresses  whether  Customs’ 
latest  plan  satisfies  these 
conditions  and  provides 
observations  about  the  plan  and 
Customs’  efforts  to  implement 
GAO’s  open  recommendations 
for  improving  ACE  management. 


What  GAO  Recommends 

Because  GAO  has  an  open 
recommendation  to  Customs  for 
correcting  its  cost  estimating 
deficiencies,  which  Customs 
reports  will  be  fully  implemented 
in  December  2002,  GAO  is  not 
making  another  recommendation 
at  this  time.  Customs  concurred 
with  GAO’s  report  and  described 
actions  that  it  is  taking  to  improve 
its  expenditure  plan  cost 
estimating. 


What  GAO  Found 

Customs’  May  2002  ACE  expenditure  plan  is  the  third  in  a  series  of 
legislatively  required  plans.  This  plan  provides  for  the  design, 
development,  and  deployment  of  the  second  release  of  the  first  of  four 
planned  ACE  increments.  The  plan  also  meets  the  legislative  conditions 
governing  investment  in  ACE  that  Congress  imposed  on  Customs. 

Since  1999,  GAO  has  reported  on  Customs’  management  of  ACE  and  made 
a  series  of  recommendations  to  correct  deficiencies.  Customs  currently 
has  efforts  under  way  to  respond  to  all  of  GAO’s  recommendations.  One  of 
these  deficiencies  that  affects  the  third  expenditure  plan  is  Customs’  lack 
of  effective  cost  estimating  capabilities.  Specifically,  the  cost  estimate  in 
the  third  expenditure  plan  is  questionable  because  of  limitations  in  how  it 
was  derived  and  inconsistencies  between  this  estimate  and  an 
independent  estimate  developed  by  a  Customs  contractor.  Customs  is 
currently  determining  the  cause  of  the  variance  in  the  estimates. 

The  following  table  compares  Customs’  expenditure  plan  cost  estimate 
with  the  Software  Engineering  Institute’s  seven  questions  for  determining 
the  reliability  of  an  estimate. 


Question 

Yes 

No 

Are  the  objectives  of  the  estimate  clear  and 
correct? 

✓ 

Has  the  task  been  appropriately  sized? 

S 

Are  the  estimated  cost  and  schedule 
consistent  with  demonstrated 
accomplishments  on  other  projects? 

V 

Have  the  factors  that  affect  the  estimate 
been  identified  and  explained? 

✓ 

Have  steps  been  taken  to  ensure  the 
integrity  of  the  estimating  process? 

✓ 

Is  the  organization’s  historical  evidence 
capable  of  supporting  a  reliable  estimate? 

✓ 

Has  the  situation  changed  since  the 
estimate  was  prepared? 

V' 

This  is  a  test  for  developing  Highlights  for  a  GAO  report.  The  full  report,  including  GAO’s  objectives,  scope,  methodology,  and  analysis  is  available 
at  www.gao.gov/cgi-bin/getrpt7GAO-02-908.  For  additional  information  about  the  report,  contact  Randolph  C.  Hite  (202-512-3439).  To  provide  comments 
on  this  test  Highlights,  contact  Keith  Fultz  (202-512-3200)  or  E-mail  HighlightsTest@gao.gov. 
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United  States  General  Accounting  Office 
Washington,  D.C.  20548 


August  9,  2002 

The  Honorable  Byron  L.  Dorgan 
Chairman 

The  Honorable  Ben  Nighthorse  Campbell 
Ranking  Minority  Member 

Subcommittee  on  Treasury  and  General  Government 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Ernest  J.  Istook,  Jr. 

Chairman 

The  Honorable  Steny  H.  Hoyer 
Ranking  Minority  Member 
Subcommittee  on  Treasury,  Postal  Service, 
and  General  Government 
Committee  on  Appropriations 
House  of  Representatives 

In  May  2002,  the  U.S.  Customs  Service  submitted  to  Congress  its  third 
expenditure  plan  seeking  release  of  $190.2  million  for  its  Automated 
Commercial  Environment  (ACE)  project,  pursuant  to  Customs’  fiscal  year 
2002  appropriation.1  ACE  is  to  be  Customs’  new  import  processing  system 
and  the  first  project  under  the  Customs  Modernization  Program.  As 
required  by  the  act,  we  reviewed  the  expenditure  plan.  Our  objectives  were 
to  (1)  determine  whether  the  third  ACE  expenditure  plan  satisfies  the 
legislative  conditions,  (2)  determine  whether  the  plan  is  consistent  with 
our  open  ACE  recommendations,  and  (3)  provide  observations  about  the 
plan  and  Customs’  management  of  ACE. 

On  June  28,  2002,  we  briefed  your  offices  on  the  results  of  this  review.  This 
report  officially  transmits  the  results  of  our  work.  The  full  briefing, 
including  our  scope  and  methodology,  is  reprinted  as  appendix  I. 

Concerning  our  first  objective,  Customs’  expenditure  plan  satisfies  the 
legislative  conditions  specified  in  the  appropriations  act.  That  is,  the  plan 
provides  for  (1)  meeting  the  capital  planning  and  investment  control 
review  requirements  of  the  Office  of  Management  and  Budget  (OMB); 


Tub.  L.  No.  107-67,  115  Stat.  514,  520  (2001). 
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(2)  complying  with  Customs’  enterprise  architecture;2  and  (3)  complying 
with  federal  acquisition  rules,  requirements,  guidelines,  and  systems 
acquisition  management  practices.  Further,  the  plan  was  reviewed  and 
approved  by  the  Joint  Capital  Investment  Review  Board3  and  OMB. 

Customs  is  also  making  progress  in  addressing  our  open  recommendations, 
which  are  as  follows: 

1.  Justify  and  make  ACE  investment  decisions  incrementally. 

2.  Before  building  each  ACE  release,  certify  to  Customs’  appropriations 
subcommittees  that  Customs’  enterprise  architecture  has  been 
sufficiently  extended  and  updated. 

3.  Develop  and  implement  a  Customs  Modernization  Office  (CMO)  human 
capital  management  strategy. 

4.  Develop  and  implement  a  rigorous  and  analytically  verifiable  cost 
estimating  program  that  embodies  the  tenets  of  effective  estimating  as 
defined  in  the  institutional  and  project-specific  estimating  guidance  of 
the  Software  Engineering  Institute  (SEI). 

5.  Limit  future  expenditure  plan  requests  for  management  reserve  funds 
to  10  percent  of  the  total  funds  requested  for  the  ACE  program,  or 
adequately  justify  any  management  reserve  requests  in  excess  of  10 
percent. 

6.  Implement  certain  process  controls  embodied  primarily  in  the  second 
level  of  SEI’s  Software  Acquisition  Capability  Maturity  Model  (SA- 
CMM),4  and  by  September  30,  2002,  assess  and  report  to  Customs’ 


2An  enterprise  architecture  is  an  institutional  blueprint  for  guiding  and  constraining 
investments  in  business  process  change  and  systems. 

3To  expedite  reviews  of  ACE  expenditure  plans,  the  Customs  and  Treasury  Investment 
Review  Boards  were  consolidated  to  form  the  Joint  Capital  Investment  Review  Board, 
which  first  met  in  November  2001. 

4Capability  Maturity  ModelSM  (CMM)  is  a  service  mark  of  Carnegie  Mellon  University,  and 
CMM  is  registered  in  the  U.S.  Patent  and  Trademark  Office.  The  SA-CMM  identifies  key 
process  areas  that  are  necessary  to  effectively  manage  software-intensive  system 
acquisitions.  Level  2  is  the  second  level  of  the  SA-CMM’s  five-level  scale;  achieving  this  level 
means  that  an  organization  has  the  software  acquisition  rigor  and  discipline  to  repeat 
project  successes. 
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appropriations  subcommittees  on  the  CMO’s  software  acquisition 
process  maturity. 

7.  Address  the  risks  associated  with  the  accelerated  ACE  acquisition 
strategy  and  report  to  Customs’  appropriations  subcommittees  on  the 
strategy  and  plans  for  mitigating  the  risks  associated  with  this  strategy. 

Finally,  we  determined  that  the  cost  estimate  in  the  third  expenditure  plan 
is  questionable  because  Customs  is  still  in  the  process  of  addressing  our 
recommendation  (see  recommendation  4)  for  having  an  effective  cost 
estimating  program.  While  the  full  impact  of  the  weaknesses  in  the  process 
used  to  derive  the  estimate  was  indeterminable,  at  least  one  limitation  (an 
inflated  labor  rate)  led  to  an  overestimate  of  about  $36  million  for 
designing,  developing,  and  deploying  ACE  increment  1  release  2.  Because 
the  expenditure  plan  includes  a  management  reserve  of  10  percent  of  the 
funding  requirement,  this  means  that  the  management  reserve  is  also 
overstated  by  about  $3.6  million. 

According  to  an  ACE  prime  contractor  executive,  the  overstatement  is 
offset  by  a  corresponding  understatement  in  the  estimated  number  of  labor 
hours,  which  the  official  attributed  to  Customs’  not  having  a  complete  set 
of  requirements  at  the  time  that  the  plan’s  estimate  was  derived.  However, 
Customs  also  arranged  for  an  independent  cost  estimate  by  another 
contractor,  which  led  to  an  estimate  that  was  $77.9  million  lower  than  the 
expenditure  plan  cost  estimate.  Customs  and  support  contractor  officials 
stated  that  this  independent  estimate  is  incomplete  and  reflects  a 
misunderstanding  by  the  cost  estimate  contractor  of  the  scope  of  work  to 
be  performed.  Customs  is  currently  determining  the  cause  of  the  variance. 

Because  we  have  an  open  recommendation  to  Customs  for  correcting  its 
cost  estimating  deficiencies,  which  Customs  reports  will  be  fully 
implemented  in  December  2002,  we  are  not  making  a  recommendation  at 
this  time. 


Agency  Comments 


In  written  comments  on  a  draft  of  this  report,  the  Acting  Director,  Office  of 
Planning,  U.S.  Customs  Service,  concurred  with  the  report  and  described 
actions  that  are  being  taken  to  improve  expenditure  plan  cost  estimating. 
These  comments  are  reprinted  in  appendix  II. 
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We  are  sending  copies  of  this  report  to  the  Chairmen  and  Ranking  Minority 
Members  of  other  Senate  and  House  committees  and  subcommittees  that 
have  authorization  and  oversight  responsibilities  for  the  Customs  Service. 
We  are  also  sending  copies  to  the  Secretary  of  the  Treasury,  the 
Commissioner  of  the  Customs  Service,  and  the  Director  of  OMB.  We  also 
will  make  copies  available  to  others  upon  request.  In  addition,  the  report 
will  be  available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

Should  you  or  your  staff  have  any  questions  on  matters  discussed  in  this 
report,  please  contact  me  at  (202)  512-3439. 1  can  also  be  reached  by  E-mail 
at  HiteR@gao.gov.  Key  contributors  to  this  report  were  Mark  Bird,  Barbara 
Collier,  Tamra  Goldstein,  Randolph  Tekeley,  Scott  Pettis,  and 
Aaron  Thorne. 


Randolph  C.  Hite 

Director,  information  Technology  Architecture 
and  Systems  Issues 
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Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Customs’  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


Briefing  for  the  staffs  of  the 
Subcommittee  on  Treasury  and  General  Government 
Senate  Committee  on  Appropriations 
and 

Subcommittee  on  Treasury,  Postal  Service,  and  General  Government 
House  Committee  on  Appropriations 

June  28,  2002 
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Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Introduction 

Objectives,  Scope,  and  Methodology 

Results  in  Brief 

Background 

Results 

Conclusions 

Agency  Comments 


Briefing  Overview 
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Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


GAO 
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Introduction 


The  U.S.  Customs  Service  is  acquiring  a  new  trade  processing  system 
known  as  the  Automated  Commercial  Environment  (ACE).  ACE  is 
intended  to 


•  promote  more  efficient  movement  of  legitimate  trade  and  more 
effective  enforcement  of  trade  laws, 

•  strengthen  border  security  operations,  and 

•  include  a  single  system  interface  between  the  trade  community  and 
the  federal  government  known  as  the  International  Trade  Data 
System  (ITDS),  which  is  to  reduce  the  data  reporting  burden  placed 
on  the  trade  community  and  facilitate  the  collection  and 
dissemination  of  trade  data  for  agencies  with  trade-related  missions. 
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Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


GAO 
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Introduction  (cont.) 


Customs  is  required  in  its  appropriation1  not  to  obligate  funds  for  ACE 
until  it  submits  to  the  Congress  ACE  expenditure  plans  that 


•  meet  the  capital  planning  and  investment  control  review  requirements 
of  the  Office  of  Management  and  Budget  (OMB),  including  security 
planning; 

•  comply  with  the  Customs  enterprise  architecture; 

•  comply  with  federal  acquisition  rules,  requirements,  guidelines,  and 
systems  acquisition  management  practices; 

•  are  reviewed  and  approved  by  the  Customs  Investment  Review 
Board,  Treasury,  and  OMB;  and 

•  are  reviewed  by  GAO. 


1  Public  Law  107-67. 
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Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


GAO 

Accountability  *  Integrity  *  Reliability 


Introduction  (cont.) 


On  May  24,  2002,  Customs  submitted  its  third  ACE  expenditure  plan  to  its 
House  and  Senate  appropriations  subcommittees,  seeking  release  of  $190.2 
million  from  the  ACE  appropriation. 


Since  1999,  we  have  conducted  a  series  of  reviews  of  Customs’  ACE/ITDS 
program  and  made  recommendations  for  strengthening  program 
management. 2 


2U.S.  General  Accounting  Office,  Customs  Service  Modernization:  Management  Improvements  Needed  on  High-Risk 
Automated  Commercial  Environment  Project,  GAO-02-545  (Washington,  D.C.:  May  13,  2002);  Customs  Service 
Modernization:  Results  of  Review  of  First  Automated  Commercial  Environment  Expenditure  Plan,  GAO-Ol-696 
(Washington,  D.C.:  June  5,  2001);  Customs  Service  Modernization:  Serious  Management  and  Technical  Weaknesses  Must 
Be  Corrected,  GAO/AIMD-99-41  (Washington,  D.C.:  Feb.  26,  1999). 
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Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


— — ^  ^  ^  Objectives,  Scope,  and  Methodology 

Accountability  *  Integrity  *  Reliability  **  7  Mr  7  cy*7 


Objectives 

As  agreed,  our  objectives  were  to 

•  determine  whether  the  third  ACE  expenditure  plan  satisfies  the 
legislative  conditions, 

•  determine  whether  the  plan  is  consistent  with  our  open  ACE 
recommendations,  and 

•  provide  observations  about  the  plan  and  Customs’  management  of 
ACE. 


Scope  and  Methodology 

To  accomplish  our  objectives,  we 

•  analyzed  the  plan  and  supporting  documents,  including  security 
plans,  to  determine  if  the  legislative  conditions  were  met; 

•  analyzed  the  plan  and  supporting  documents  to  determine 
consistency  of  actions  taken  with  our  open  recommendations; 
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Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Objectives,  Scope,  and  Methodology  (cont.) 


Scope  and  Methodology  (cont.) 


•  analyzed  progress  and  plans  for  integrating  ACE  and  ITDS; 

•  interviewed  Customs,  ITDS,  OMB,  and  contractor  officials; 

•  analyzed  ACE  and  ITDS  supporting  documentation  to  clarify  and 
verify  information  in  the  expenditure  plan;  and 

•  as  agreed,  did  not  independently  validate  Customs’  provided 
information,  including  cost  estimates  and  project  content  and  status 
information. 


We  performed  our  work  from  April  2002  through  June  2002  in  accordance 
with  generally  accepted  government  auditing  standards. 
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Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Results  in  Brief;  Objective  1 


Customs’  expenditure  plan  satisfies  the  legislative  conditions. 


Legislative  conditions 

Satisfies 

Does  not 
satisfy 

1. 

Meets  OMB  capital  planning  and  investment  control  review 
requirements,  including  OMB  Circular  A-ll,  part  33 

S 

2. 

Complies  with  the  Customs  enterprise  architecture 

S 

3. 

Complies  with  federal  acquisition  rules,  requirements, 
guidelines,  and  systems  acquisition  management  practices 

S 

4. 

Is  reviewed  and  approved  by  the  Customs  Investment 
Review  Board,  Treasury,  and  OMB 

S 

5. 

Is  reviewed  by  GAO 

S 

Customs  provided  us  with  a  draft  of  the  expenditure  plan  before  it  was  submitted  to  the  appropriations  subcommittees. 
We  determined  that  the  draft  plan  did  not  satisfy  the  requirements  of  OMB  Circular  A-ll,  part  3.  After  we  discussed  our 
analysis  with  Customs  and  support  contractor  officials,  Customs  amended  the  expenditure  plan  to  satisfy  the  requirements 
before  submitting  it  to  the  appropriations  subcommittees. 
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Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


Customs’  expenditure  plan  is  consistent  with  our  open  recommendations.4 


Open  recommendations 

Consistent5 
Complete  |  In  progress 

Inconsistent 

1.  Justify  and  make  investment  decisions  incrementally. 

2.  Before  building  each  ACE  release,  certify  to  Customs’ 
appropriations  subcommittees  that  Customs’ 
enterprise  architecture  has  been  sufficiently  extended 
and  updated. 

3.  Develop  and  implement  a  rigorous  and  analytically 
verifiable  cost  estimating  program  that  embodies  the 
tenets  of  effective  estimating  as  defined  in  the  Software 
Engineering  Institute’s  (SEI)  institutional  and  project- 
specific  estimating  guidance. 

4.  Immediately  develop  and  implement  a  Customs 
Modernization  Office  (CMO)  human  capital 
management  strategy. 

4  Because  Customs  developed  its  third  expenditure  plan  during  our  review  of  Customs’  second  expenditure  plan,  Customs 
is  in  the  process  of  addressing  the  open  recommendations  resulting  from  our  review  of  Customs’  second  expenditure  plan. 
’’Complete  means  that  Customs  has  addressed  the  recommendation  and  no  additional  action  by  Customs  is  required;  in 
progress  means  either  that  Customs  has  taken  steps  to  begin  addressing  the  recommendation  or  that  Customs  has  taken 
such  steps,  but  additional  steps  are  needed  because  the  recommendation  concerns  a  recurring  issue. 
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Appendix  I 
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Results  in  Brief:  Objective  2  (cont.) 


Open  recommendations 


Consistent 

Complete  In  progress 


Inconsistent 


5.  Limit  future  expenditure  plan  requests  for  management 
reserve  funds  to  10  percent  of  the  total  funds  requested  for 
the  program,  or  adequately  justify  any  management 
reserve  requests  in  excess  of  10  percent. 

6.  Implement  process  controls  for  SEI’s  Software  Acquisition 
Capability  Maturity  Model  (SA-CMM), 11  and  by  September 

30,  2002,  assess  and  report  to  Customs’  appropriations 
subcommittees  on  the  CMO’s  software  acquisition  process 
maturity. 

V 

7.  Address  the  risks  associated  with  the  accelerated  ACE 
acquisition  strategy  and  report  on  the  strategy  going 
forward  and  plans  for  mitigating  the  risks  associated  with 
this  strategy. 

Capability  Maturity  Model  is  a  service  mark  of  Carnegie  Mellon  University,  and  CMM  is  registered  in  the  U.S.  Patent  and 
Trademark  Office.  The  SA-CMM  identifies  key  process  areas  that  are  necessary  to  effectively  manage  software-intensive 
system  acquisitions.  Level  2  is  the  second  level  of  the  SA-CMM’s  five-level  scale;  achieving  this  level  means  that  an 
organization  has  the  software  acquisition  rigor  and  discipline  to  repeat  project  successes. 
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Results  in  Brief;  Objective  3 


Other  observation: 


ACE  cost  estimate 


The  cost  estimate  in  the  third  expenditure  plan  is  questionable 
because  Customs  is  still  in  the  process  of  addressing  our  open 
recommendation  for  having  an  effective  cost  estimating  program. 
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ACE/ITDS  Overview 


Background 


ACE  is  a  Customs  Service  import  processing  system  that  is  expected  to 
support  fundamental  improvements  in  the  service’s  trade  compliance 
mission  area.  ACE  is  to  provide,  for  example, 

•  more  efficient  import  and  export  operations, 

•  increased  legitimate  cargo  flow, 

•  reduced  paper  burden, 

•  more  efficient  and  effective  account  management,  and 

•  improved  data  analysis  for  trade  law  enforcement  and  border 
security. 


ITDS  is  a  govemmentwide  system  that  is  intended  to  provide  a  single 
system  interface  between  the  trade  community  and  federal  agencies  with 
trade-related  missions.  ITDS  is  to 

•  reduce  the  data  reporting  burden  placed  on  the  trade  community  and 

•  promote  more  efficient  data  collection,  processing,  and  dissemination 
within  the  federal  government. 
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Background  (cont.) 


Customs  is  using  a  prime  contractor,  supported  by  subcontractors,  to 
develop  and  deploy  ACE/ITDS.  On  April  27,  2001,  Customs  awarded  a 
5-year,  indefinite-delivery  indefinite-quantity  contract  to  International 
Business  Machines  (IBM)  Global  Services.  IBM  and  its  subcontractors  are 
collectively  called  the  e-Customs  Partnership  (eCP). 


Customs’  current  plans  call  for  release  of  ACE/ITDS  capabilities  in  four 
increments  over  4  years.  Each  increment  contains  discrete  functionality, 
building  on  the  functionality  from  previous  increments.  The  first  three 
increments  are  further  divided  into  two  releases. 


The  following  figure  and  table  illustrate,  for  each  ACE  increment  and 
release,  (1)  the  functions,  (2)  users,  and  (3)  locations  by  transportation 
modes. 
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Background  (cont.) 


Allocation  of  ACE/ITDS  Functions  across  Increments 


Functions 

Release  1 

ACE  increment' 

l  2  3 

Release  2 

4 

Enterprise  portal  and  IT  infrastructure 

Initial 

Additional 

Additional 

Additional 

Remaining 

Data  processing  (e.g.,  reference  library  and  data  query 
capabilities) 

Initial 

Additional 

Additional 

Additional 

Remaining 

Trade  agreement  (e.g.,  manage  visas,  quotas;  implement 
trade  agreements) 

Initial 

Remaining 

Integration  of  ITDS  functionality 

Initial 

Additional 

Additional 

Remaining 

Account  management  (e.g.,  create,  access  accounts) 

Initial 

Additional 

Additional 

Remaining 

Account  revenue  (e.g.,  generate  billing  statements, 
accept  payments) 

Initial 

Additional 

Additional 

Remaining 

Enforcement  (e.g.,  determine  violation  severity,  resolve 
compliance  violations) 

Initial 

Additional 

Remaining 

Selectivity  criteria  for  assessing  noncompliance  risk, 
screening  incoming  transactions 

Initial 

Additional 

Remaining 

Cargo  release  and  tracking 

Initial 

Additional 

Additional 

Remaining 

Ability  to  process  multi-modal  goods,  full  export 
functions,  decommissioning  of  ACS8 

All 

'  As  of  June  2002,  only  Increment  1  functionality  had  been  allocated  to  releases.  Future  release  allocation  is  expected  for 
Increments  2  and  3.  All  means  all  capabilities  within  this  function  are  provided  in  this  increment  or  release;  initial  means  a 
beginning  set  of  capabilities  is  provided,  and  the  number  and  type  of  users  may  be  limited  for  certain  capabilities; 
additional  means  an  increased  set  of  capabilities  is  provided,  and  the  number  and  type  of  users  may  be  expanded  for 
specific  capabilities;  remaining  means  that  the  final  set  of  capabilities  is  provided  to  intended  users. 

8  The  Automated  Commercial  System  (ACS)  tracks,  controls,  and  processes  all  goods  imported  into  the  United  States.  ACE 
is  expected  to  provide  all  fimctions  currently  provided  by  ACS. 
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Background  (cont.) 


Delivery  of  ACE/ITDS  Incremental  Capabilities  to  Users9 


ACE/ITDS 

increment 

Customs 

Other  agencies 

Trade/foreign  governments 

1.1 

1500  employees  with 
Web-capable 
workstations, 
including  25  national 
account  managers10 

Initial  account  management  functions  to  40 
importers  and  their  associated  brokers  and 
carriers. 

1.2 

All  primary  and 
secondary 
inspectors11  at  truck 
processing  locations 

Cargo  release  referral  data  to 

Department  of  Transportation’s  Federal 
Motor  Carrier  Safety  Administration 
(FMCSA) 

Expand  account  management  functions  to  1100 
importers 

2.1 

Access  to  transaction  data  and 
conveyance,  crew,  and  cargo  release 
data  for  four  federal  agencies12 

Expand  account  to  high-volume,  low-risk 
importers,  exporters,  and  their  associated 
brokers  and  carriers 

2.2 

Extend  access  to  additional  federal 
agencies 

Extend  account  functionality  to  major  importers 
and  exporters,  and  their  carriers  and  brokers 

3 

Extend  access  to  additional  federal 
agencies 

Provide  foreign  governments  access  to  visa  and 
quota  data 

4 

Extend  access  to  additional  federal 
agencies 

Expand  foreign  governments’  access  to  visa, 
quota,  license,  and  permit  data 

9  As  of  June  2002,  only  the  number  and  type  of  users  for  Increment  1  had  been  defined.  The  specific  number  and  types  of 
users  for  Increments  2,  3,  and  4  are  expected  to  be  defined  at  a  future  date. 

19  National  account  managers  are  25  Customs  employees  assigned  to  work  with  the  largest  importers  to  ensure  their 
compliance  with  trade  laws. 

11  Primary  inspectors  perform  brief  inspections  of  passenger  vehicles  and,  for  trucks,  examinations  of  documentation  and 
cargo,  driver,  and  truck.  Secondary  inspectors  perform  more  detailed  inspections  as  a  result  of  the  primary  inspection. 

12  Food  and  Drug  Administration  (Center  for  Food  Safety  and  Nutrition),  Department  of  Agriculture  (Animal  and  Plant 
Health  Inspection  Service),  Immigration  and  Naturalization  Service  (crew  release  data  only),  and  Census  Bureau. 
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Background  (cont.) 


Delivery  of  ACE/ITDS  Increments  to  Locations  by  Mode 


ACE/ITDS  increment 

Location  by  mode 

1 

All  truck  ports 

2 

All  rail  ports 

All  airports 

All  seaports 

3 

All  rail  ports 

All  airports 

All  seaports 

All  ports  supporting  mail  entry 

4 

All  truck  ports 

All  rail  ports 

All  airports 

All  seaports 

All  ports  supporting  mail  entry 

Each  of  Customs’  304  ports  process  multiple  modes  of  transportation: 

•  103  locations  process  trucks, 

•  172  locations  process  aircraft, 

•  33  locations  process  trains, 

•  109  locations  process  ships,  and 

•  15  locations  process  mail. 
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Background  (cont.) 


•  Detailed  Description  of  Increment  1  Release  2 


The  $190.2  million  requested  in  the  third  ACE  expenditure  plan  is  for 
designing,  developing,  and  deploying  increment  1  release  2. 


Building  on  increment  1  release  1  (which  is  to  include  infrastructure 
improvements — such  as  a  Web  portal13 — and  initial  functionality — such  as 
account  access),  release  2  is  to  support 


•  enhanced  account  management  by  providing  the  ability  to  create, 
manage,  and  assess  accounts  for  compliance; 

•  initial  cargo  release  functions,  including  the  ability  to  expedite  the 
release  of  highly  compliant  importers  transporting  goods  via  trucks; 

•  initial  enforcement  functions,  including  the  ability  to  create  historical 
records  of  interactions  between  an  account  and  Customs;  and 

•  initial  account  revenue  functions,  including  the  ability  to  issue 
periodic  billing  statements  and  accept  electronic  payment. 

Release  2  is  to  be  deployed  at  all  truck  processing  ports,  and  is  to  support 
1500  Customs  personnel,  1100  importers,  and  FMCSA. 


13  A  Web  portal  is  an  entry  point  on  the  Internet  (Web  site)  that  offers  such  resources  and  services  as  search  engines  and 
other  on-line  capabilities. 
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Background  (cont.) 


Summary  of  Past  GAO  Reviews  on  ACE/ITDS  Integration 


Before  1999,  ITDS  management  responsibility  resided  in  the  ITDS  Board  of 
Directors.  ITDS  also  had  a  program  office  that  resided  in  the  Department 
of  the  Treasury.  At  that  time,  ACE  program  management  responsibility 
resided  in  Customs.  In  February  1999,  we  reported  on  overlap,  duplication, 
and  inconsistencies  between  the  two  systems,  and  we  recommended, 
among  other  things,  that  Customs  consider  satisfying  some  ACE 
requirements  through  ITDS.14  Subsequently,  Treasury  transferred  program 
management  responsibility  for  ITDS  to  Customs  to  better  integrate  the  two 
systems. 

In  April  2001  we  briefed  your  committees  on  the  results  of  our  second 
review  of  the  ACE  program,  noting,  among  other  things,  that  Customs  was 
investing  in  an  ITDS  pilot  project  separate  from  the  ACE  program.  We 
recommended  steps  to  integrate  ACE  and  ITDS  management  and 
technology.  Subsequently,  Customs  transferred  ITDS  program 
management  responsibility  to  the  Customs  Modernization  Office  (CMO) 
and  committed  to  an  integrated  approach  for  managing  ACE  and  ITDS. 


“U.S.  General  Accounting  Office,  Customs  Service  Modernization:  Serious  Management  and  Technical  Weaknesses  Must 
Be  Corrected,  GAO/AIMD-99-41  (Washington,  D.C.:  Feb.  26,  1999). 
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Background  (cont.) 


To  this  end,  in  February  2002,  Customs  expanded  eCP’s  ITDS 
responsibilities  to  include 


•  reviewing  existing  ITDS  assets  for  use  in  ACE, 

•  enhancing  the  governance  structure  for  the  ITDS  Board  of  Directors 
to  increase  the  effectiveness  of  interagency  cooperation,  and 

•  modifying  Customs’  enterprise  architecture  as  necessary  to  ensure 
that  it  is  consistent  with  ACE  and  ITDS  architectures. 


Overview  of  ACE/ITDS  System  and  Users 

The  following  figure  provides  a  simplified  representation  of  the  functional 
integration  of  ACE  and  ITDS,  and  the  integrated  system’s  relationship  with 
the  trade  community  and  federal  agencies  with  trade-related  missions.10 


15  There  are  104  federal  agencies  that  have  trade-related  missions.  The  ITDS  Board  has  identified  29  of  them  as  major 
international  trade  players. 
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Simplified  Overview  of  ACE/ITDS  and  Users 


Background  (cont.) 


Trade  community 


Other  federal  agencies 
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Background  (cont.) 


Chronology  of  ACE  Expenditure  Plans  and  Funding 


In  March  2001,  Congress  released  to  Customs  $5  million  in  funding  for  the 
ACE  program  to  sustain  Customs  modernization  office  operations. 


On  March  26,  2001,  Customs  submitted  its  first  ACE  expenditure  plan  to 
Congress  for  $45  million  to  sustain  modernization  operations,  develop  top- 
level  ACE  requirements,  and  provide  the  infrastructure  needed  for 
management  of  the  modernization  program. 


On  February  1,  2002,  Customs  submitted  its  second  ACE  expenditure  plan 
to  Congress  for  $206.9  million  to  sustain  modernization  operations;  design, 
develop,  and  deploy  increment  1  release  1;  and  develop  increment  2 
requirements. 

Congress  subsequently  approved  the  use  of  $188.6  million,  bringing  total 
ACE  funding  to  $238.6  million. 


On  May  24,  2002,  Customs  submitted  its  third  ACE  expenditure  plan  to 
Congress,  seeking  release  of  $190.2  million  from  the  ACE  appropriation  to 
finance  the  design,  development,  and  implementation  of  ACE  increment  1 
release  2. 
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Background  (cont.) 


The  following  figures  illustrate  the  recent  history  of  ACE  appropriations, 
expenditure  plans  and  funding  history,  and  the  funding  relationships 
among  ACE  expenditure  plans  and  system  increments. 
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Background  (cont.) 


ACE  funding  releases  shown  by  fiscal  year  appropriations 


FY  2001  appropriations  =  $130  million 


FY  2002  appropriations  =  $300  million 


March  2001 
$5  million 
funding 
release 


April  2001 
$45  million 
funding 
release 


March  2002 
$188.6 
million 
funding 
release 


$108.9  million 


$79.7  million 


$190.2  million 


1.9  million 


Next  expenditure 
plan  (planned  for 
October  2002) 


Third  expenditure  plan 
for  $190.2  million 
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Accountability  *  Integrity  *  Reliability  ^  ' 


Illustrated  chronology  of  ACE  expenditure  plans  and  funding 


Cumulative  funding  through  second  expenditure  plan  =  $238.6  million. 
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Background  (cont.) 


Mapping  of  ACE  increments  and  releases  to  expenditure  plans 


jRequirem  ents  &  planning 

^Design  (preliminary  &  detailed),  develop,  deploy 
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Legislative  conditions 

Satisfies? 

Results  of  our  analysis 

1.  Plan  meets  OMB 
capital  planning  and 
investment  control 
review  requirements, 
including  OMB 

Circular  A-ll,  part  3. 

The  expenditure  plan  and  supporting 
documentation  address  OMB  capital  planning  and 
investment  control  review  requirements,  including 
OMB  circular  A-ll,  part  3.  For  example,  the 
expenditure  plan  addresses  key  requirements  such 
as  compliance  with 

•  federal  security  and  privacy  regulations; 

•  Government  Paperwork  Elimination  Act 
requirements;  and 

•  strategic  planning  requirements  that 
demonstrate  how  the  project  will  use  a 
performance-based  measurement  system. 

2.  Plan  complies  with 
Customs’  enterprise 
architecture. 

The  expenditure  plan  is  for  the  design, 
development,  and  implementation  of  ACE 
increment  1  release  2.  Customs  plans  to  certify  that 
ACE  increment  1  release  2  is  compliant  with 

Customs’  enterprise  architecture  before  beginning 
detailed  design  activities. 
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Results:  Objective  1  (cont.) 


Legislative  conditions 

Satisfies? 

Results  of  our  analysis 

3.  Plan  complies  with 
federal  acquisition 
rules,  requirements, 
guidelines,  and 
systems  acquisition 
management 
practices. 

V 

The  expenditure  plan  provides  for  satisfying 
acquisition  practices.  For  example,  it  includes  steps 
for  implementing  the  SEI  SA-CMM  key  process 
areas  and  using  Customs’  investment  management 
process  to  ensure  that  the  ACE  acquisition  will 
comply  with  appropriate  acquisition  practices.  If 
these  processes  are  implemented  effectively, 
Customs’  processes  should  meet  federal  acquisition 
rules,  requirements,  guidelines,  and  systems 
acquisition  management  practices. 

4.  Plan  is  reviewed  and 
approved  by  Customs 
Investment  Review 
Board,  Treasury,  and 
OMB. 

V 

The  Customs  and  Treasury  Investment  Review 
Boards  were  consolidated  to  form  the  Joint  Capital 
Investment  Review  Board,  which  approved  the 
expenditure  plan  on  March  7,  2002.  OMB  approved 
the  plan  on  April  30,  2002. 

5.  Plan  is  reviewed  by 
GAO. 

V 

Customs  delivered  its  expenditure  plan  to  Congress 
on  May  24,  2002.  Our  review  was  completed  on  June 
24,  2002. 
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Customs’  expenditure  plan  is  consistent  with  our  open  recommendations. 


Open  recommendation 


Justify  and  make  investment  decisions 
incrementally,  and  for  each  increment, 


use  disciplined  investment 
processes; 


prepare  realistic  and  supportable 
benefit  expectations; 


require  a  favorable  return  on 
investment  and  compliance  with 
Customs  architecture  before 
making  an  investment  decision; 
and 

validate  actual  costs  and  benefits 
once  an  increment  is  piloted. 


Results  of  our  analysis 


Customs  has  committed  to  and  defined  process 
controls  for  justifying  and  making  investment 
decisions  incrementally.  For  ACE  increment  1 
release  2,  Customs 

•  followed  its  investment  management 
process,  which  we  determined  meets 
OMB’s  policy  for  management  of  federal 
information  resources; 

•  prepared  and  is  in  the  process  of  refining 
benefit  expectations  for  ACE  increment  1, 
which  we  determined  are  generally 
consistent  with  OMB’s  policy  for  cost 
benefit  analyses  of  federal  programs; 

•  reported  a  favorable  return  on  investment 
and  plans  to  certify  its  compliance  with 
Customs  architecture  before  beginning 
detailed  design  activities;  and 

•  plans  to  validate  actual  costs  and  benefits 
as  determined  in  post-implementation 
reviews. 


Page  32 


GAO-02-908  Customs  Service  Modernization 


Appendix  I 

Custom’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


Open  recommendation 

Results  of  our  analysis 

Before  building  each  ACE  release  (i.e., 
beginning  detailed  design  and 
development),  certify  to  Customs’ 

House  and  Senate  appropriations 
subcommittees  that  the  enterprise 
architecture  has  been  sufficiently 
extended  to  provide  the  requisite 
enterprise  design  content  and  has  been 
updated  to  ensure  consistency  and 
integration  across  business  areas. 

Before  building  each  ACE  increment  (i.e., 
beginning  detailed  design  and  development), 
Customs  plans  to  certify  to  its  House  and  Senate 
appropriations  subcommittees  that  the  enterprise 
architecture  has  been  sufficiently  extended  to 
provide  the  requisite  enterprise  design  content 
and  has  been  updated  to  ensure  consistency  and 
integration  across  business  areas.  Customs  plans 
the  first  certification  of  its  enterprise  architecture 
before  building  ACE  increment  1  release  1. 

Open  recommendation 

Results  of  our  analysis 

Develop  and  implement  a  rigorous  and 
analytically  verifiable  cost  estimating 
program  that  embodies  the  tenets  of 
effective  estimating  as  defined  in  SEI’s 
institutional  and  project-specific 
estimating  models. 

By  December  2002,  Customs  plans  to  develop  and 
implement  a  cost  estimating  program  that 
employs  the  tenets  of  effective  cost  estimating  as 
defined  in  SEI’s  cost  estimating  guidelines. 
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Results:  Objective  2  (cont.) 


Open  recommendation 


Results  of  our  analysis 


Immediately  develop  and  implement  a 
CMO  human  capital  management 
strategy  that  provides  both  near-term 
and  long-term  solutions  to  the  CMO’s 
human  capital  capacity  limitations, 
including  (1)  defining  the  office’s  skill 
and  capability  needs  in  terms  that  will 
allow  Customs  to  attract  qualified 
individuals  and  (2)  providing  sufficient 
rewards  and  training,  linked  to 
performance,  to  promote  their 
retention. 


Customs  plans  to  develop  a  human  capital 
management  strategy.  Thus  far,  Customs  has 
increased  the  number  of  positions  in  the  CMO 
from  11  to  23  and  raised  their  grade  levels  to 
address  immediate  staffing  shortfalls  (a  draft 
structure  has  been  defined  and  is  awaiting 
Commissioner  approval).  According  to  Customs 
officials,  the  strategy  will  describe  the  CMO’s 
skills  and  capability  needs  (core  competencies) 
linked  to  performance  and  reward  programs  for 
each  of  the  23  positions,  and  will  define  training 
for  these  positions.  Customs  plans  to  complete 
the  initial  strategy  by  September  30,  2002,  and 
revise  it  periodically. 
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Open  recommendation 

Results  of  our  analysis 

Limit  future  expenditure  plan  requests  for 
management  reserve  funds  to  10  percent 
of  the  total  funds  requested  for  the 
program,  or  adequately  justify  any 
management  reserve  requests  in  excess  of 
10  percent. 

Customs’  third  expenditure  plan  limits  its 
request  for  management  reserve  to  10  percent 
of  the  total  funds  requested.  Customs  plans  to 
limit  future  requests  for  management  reserve  to 
10  percent  or  provide  adequate  justification  for 
any  management  reserve  request  that  exceeds 

10  percent. 

Open  recommendation 

Results  of  our  analysis 

Develop  and  implement  process  controls 
for  the  SEI  SA-CMM  level  2  key  process 
areas  and  for  the  level  3  acquisition  risk 
management  key  process  area,  and  by 
September  30,  2002,  conduct  and  report 
to  Customs’  House  and  Senate 
appropriations  subcommittees  on  the 
results  of  an  internal  assessment  of  the 
CMO’s  maturity  against  the  SEI  SA-CMM 
level  2  key  process  areas  and  the  level  3 
acquisition  risk  management  key  process 
area. 

Customs  plans  to  have  all  recommended  SEI 
SA-CMM  process  controls  in  place  by 

September  30,  2002.  At  that  time,  Customs 
plans  to  conduct  an  internal  assessment  of  the 
CMO’s  maturity  against  SA-CMM  level  2  key 
process  areas  and  the  level  3  acquisition  risk 
management  key  process  area  and  report  the 
results  to  Customs’  House  and  Senate 
appropriations  subcommittees. 
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Results:  Objective  2  (cont.) 


Open  recommendation 


Results  of  our  analysis 


Address  the  risks  associated  with  the 
accelerated  ACE  acquisition  strategy, 
including 

(1)  immediately  analyzing  the  risks 
associated  with  the  degree  of 
design,  development,  and  testing 
concurrency  across  ACE 
increments  that  is  inherent  in 
Customs’  4-year,  schedule-driven 
acquisition  strategy; 


Customs  addressed  and  has  committed  to 
continuously  addressing  the  risks  associated  with 
the  accelerated  ACE  acquisition  strategy.  As  a 
first  step,  Customs  has 

(1)  analyzed  the  risks  by 

•  quantifying  the  costs  associated  with  the 
following  risks: 

-  rework  as  a  result  of  parallel  design  and 
development  tasks; 

-  slower  than  predicted  adoption  of  ACE 
functionality;  and 

-  increased  cost  of  additional  staff  for 
multiple  concurrent  tasks; 

•  comparing  the  cost  of  the  4-  and  5-year 
schedules;  and 

•  identifying  qualitative  benefits  associated 
with  the  4-year  schedule  (e.g.,  providing 
functionality  earlier  to  Customs;  providing 
benefits  earlier  to  the  trade  community  and 
other  federal  agencies;  and  supporting 
border  security  activities). 
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Results:  Objective  2  (cont.) 


Open  recommendation 


(2)  reconsidering  the  merits  of  this 
accelerated  strategy;  and 


Results  of  our  analysis 


(2)  presented  the  results  of  the  analysis  to  the 

Customs’  Executive  Steering  Committee,  which 
showed  that  while  quantifiable  return  on 
investment  from  the  accelerated  schedule  was 
marginal,  the  qualitative  benefits  associated 
with  stronger  border  security  justified  pursuing 
the  accelerated  schedule;  and 


(3)  within  90  days  of  the  date  of  this 
briefing,  reporting  to  Customs’ 
House  and  Senate  appropriations 
subcommittees  on  the  agency’s 
strategy  going  forward  and  its 
plans  for  mitigating  the  inherent 
risks  associated  with  this  strategy. 


(3)  on  May  13,  2002,  reported  to  Customs’  House 
and  Senate  appropriations  subcommittees  on 
the  agency’s  decision  to  implement  the  4-year 
schedule,  but  commits  to  not  being  schedule 
driven  in  acquiring  and  deploying  ACE/ITDS 
and  to  making  risk-based  decisions  on  how  to 
proceed.  Customs  supporting  documentation, 
including  the  ACE  Program  Plan,  identifies 
associated  program  risks  and  proposed 
mitigation  techniques  for  each  of  these  risks. 
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Results;  Objective  3 


Observation:  The  cost  estimate  in  the  third  expenditure  plan  is 
questionable  because  Customs  is  still  in  the  process  of  addressing 
our  open  recommendation  for  having  an  effective  cost  estimating 
program. 


As  we  have  previously  reported,  the  ability  to  produce  reliable  cost 
estimates  for  inclusion  in  ACE  expenditure  plans  is  essential.  Without  such 
estimates,  Customs  cannot  provide  congressional  committees  with  the 
information  needed  to  exercise  important  program  oversight.  Ensuring  that 
cost  estimates  are  reliable  requires,  at  a  minimum,  that  Customs  satisfy 
itself  that  rigorous,  analytically  verifiable  methods  are  used  to  derive  the 
estimates. 
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Results:  Objective  3  (cont.) 


SEI  has  defined  guidance  for  validating  the  reliability  of  an  estimate.11’  This 
guidance  identifies  seven  questions.  Each  question  is  associated  with 
elements  that,  if  present,  support  the  credibility  of  the  estimate. 


Seven  Questions  to  Ask  to  Determine  Reliability  of  Project  Estimates 


1.  Are  the  objectives  of  the  estimate  clear  and  correct? 

2.  Has  the  task  been  appropriately  sized? 

3.  Are  the  estimated  cost  and  schedule  consistent  with  demonstrated 
accomplishments  on  other  projects? 

4.  Have  the  factors  that  affect  the  estimate  been  identified  and 
explained? 

5.  Have  steps  been  taken  to  ensure  the  integrity  of  the  estimating 
process? 

6.  Is  the  organization’s  historical  evidence  capable  of  supporting  a 
reliable  estimate? 

7.  Has  the  situation  changed  since  the  estimate  was  prepared? 


16  A  Manager’s  Checklist  for  Validating  Software  Cost  and  Schedule  Estimates  (CMU/SEI-95-SR-004). 
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Results:  Objective  3  (cont.) 


As  stated  earlier,  Customs  plans  to  develop  a  cost  estimating  program  by 
December  2002.  For  its  third  expenditure  plan,  however,  Customs  used  a 
cost  estimate  prepared  by  its  prime  contractor.  Customs  subsequently 
tasked  its  support  contractor  with  validating  the  cost  estimate.  Using  the 
SEI  guidance,  the  support  and  prime  contractors  identified  weaknesses  and 
reasons  for  these  weaknesses,  shown  in  the  table  below. 


SEI  question 

Yes 

No 

Weakness/reason 

1.  Art'  the  objectives  of  the 
estimate  clear  and  correct? 

2.  Has  the  task  been 

appropriately  sized?  (e.g., 
processes  for  estimating 
size  are  documented) 

The  prime  contractor  did  not  completely 
document  its  processes  for  estimating  size  due  to 
limited  time  available  in  the  schedule.  In  addition 
to  this  weakness  identified  by  Customs’  support 
contractor,  our  analysis  determined  that  the 
individual  who  estimated  size  was  not  certified  in 
function  point  analysis,  the  sizing  methodology 
used.  Certification  in  the  methodology  better 
ensures  an  objective,  comparative  measurement 
that  is  consistent  and  accurate.1' 

17  The  International  Function  Point  Users’  Group  maintains  the  Function  Point  Counting  Practices  Manual,  the  recognized 
industry  standard  for  function  point  analysis,  which  is  a  methodology  for  software  sizing.  The  group  offers  professional 
certification  for  practitioners  of  function  point  analysis  as  Certified  Function  Point  Specialists. 
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Results:  Objective  3  (cont.) 


SEI  question 

Yes 

No 

Weakness/reason 

3.  Are  the  estimated  cost  and 
schedule  consistent  with 
demonstrated 
accomplishments  on  other 
projects? 

Estimated  cost  and  schedule  consistency  with 
demonstrated  accomplishments  on  other 
projects  was  not  determined.  Although  the  prime 
contractor  had  cost  and  schedule  data  for  other 
projects,  the  contractor  decided  not  to  use  these 
data  because  the  projects  were  not  part  of 
Customs’  modernization  program. 

4.  Have  the  factors  that  affect 
the  estimate  been  identified 
and  explained?  (e.g.,  a  risk 
analysis  has  been 
performed) 

The  prime  contractor  did  not  perform  a  risk 
analysis  as  part  of  the  cost  model  due  to,  among 
other  things,  limitations  in  the  available  schedule; 
instead  the  prime  contractor  used  subjectively 
adjusted  data  (high  labor  rate,  low  COTSlf> 
utilization  rate,  and  low  developer  productivity 
rate)  in  an  attempt  to  address  an  undefined  level 
of  risk. 

5.  Have  steps  been  taken  to 
ensure  the  integrity  of  the 
estimating  process?  (e.g., 
more  than  one  cost  model 
or  estimating  approach  has 

The  prime  contractor  did  not  use  more  than  one 
cost  model  due  to  limitations  in  the  available 
schedule. 

18  Commercial,  off  the  shelf  (software  or  hardware). 
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Results:  Objective  3  (cont.) 


SEI  question 

Yes 

No 

Weakness/reason 

been  used) 

6.  Is  the  organization’s 

historical  evidence  capable 
of  supporting  a  reliable 
estimate? 

Although  the  prime  contractor  had  historic 
evidence  for  other  projects,  the  contractor 
decided  not  to  use  this  evidence  because  the 
projects  were  not  part  of  Customs’ 
modernization  program. 

7.  Has  the  situation  changed 
since  the  estimate  was 
prepared? 

Yes  and  the  expenditure  plan  estimate  was  not 
changed  to  reflect  the  changed  situation. 
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Results:  Objective  3  (cont.) 


While  the  full  impact  of  these  weaknesses  has  not  been  determined,  at 
least  one  limitation  (inflated  labor  rate)  led  to  a  quantifiable  estimate 
overstatement.  Specifically,  an  $8000  per  person  month  labor  rate 
variance  resulted  in  a  funding  requirement  overestimate  of  about  $36 
million  for  designing,  developing,  and  deploying  increment  1  release  2. 
Because  the  expenditure  plan  includes  a  management  reserve  of  10 
percent  of  the  funding  requirement,  this  means  that  the  management 
reserve  is  also  overstated  by  $3.6  million. 

According  to  the  eCP  Program  Executive,  the  $8000  per  person  month 
overstatement  is  offset  by  a  corresponding  understatement  in  the 
estimated  number  of  labor  hours,  which  the  official  attributed  to  the  lack 
of  a  complete  set  of  requirements  at  the  time  that  the  plan’s  estimate  was 
derived. 


In  contrast,  Customs’  cost  estimate  contractor  performed  an  independent 
estimate  of  the  labor  cost  for  executing  the  third  expenditure  plan.  The 
estimate  was  $77.9  million  lower  than  the  labor  estimate  contained  in  the 
expenditure  plan.  Customs  and  support  contractor  officials  stated  that  the 
independent  estimate  is  not  complete  and  reflects  a  misunderstanding  by 
the  estimator  of  the  scope  of  work  to  be  performed,  and  that  Customs  is 
currently  determining  the  root  cause  of  the  variance. 
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Conclusions 


Customs’  third  ACE  expenditure  plan  satisfies  the  legislative  conditions 
and  is  consistent  with  our  open  recommendations.  However,  the  cost 
estimate  in  the  plan  is  questionable. 
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Agency  Comments 


We  provided  a  draft  of  this  briefing  and  discussed  its  contents  with 
Customs’  Assistant  Commissioner  for  Information  and  Technology,  the 
Chairman  of  the  ITDS  Board  of  Directors,  the  eCP  Program  Executive,  and 
Customs’  support  contractor  officials.  The  officials  agreed  with  our 
findings  and  conclusions  that  Customs’  third  ACE  expenditure  plan 
satisfies  the  legislative  conditions  and  is  consistent  with  our  open 
recommendations. 


However,  Customs  and  support  contractor  officials  provided  comments 
about  our  observation  on  the  expenditure  plan  cost  estimate.  They  agreed 
that  the  preferred  approach  to  developing  an  expenditure  plan  estimate  is 
conducting  a  rigorous  analysis  of  technical,  schedule,  and  cost  estimate 
risks  and  stated  that  the  estimate  was  not  as  rigorously  developed  as  future 
estimates  will  be.  However,  they  also  stated  that  the  expenditure  plan 
estimate  remains  valid  and  that  our  assumption  that  a  $39.6  million  savings 
from  use  of  a  lower  labor  rate  is  inaccurate.  We  disagree  with  this 
statement  because  we  do  not  make  such  an  assumption.  While  we  used 
available  data  to  determine  that  the  prime  contractor’s  estimate  is 
overstated  by  $39.6  million,  we  did  not  determine  whether  the  total 
estimate  is  overstated  or  understated  because  data  were  not  available  to 
quantify  other  estimate  weaknesses  and  risk.  Thus,  we  concluded  that  the 
estimate  is  questionable. 
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Agency  Comments  (cont.) 


Customs  also  provided  new  information.  We  have  incorporated  this 
information  in  the  briefing  as  appropriate. 
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U.S.  Customs  Service 

Memorandum 


DATE:  July  31,  2002 


MEMORANDUM  FOR  RANDOLPH  C.  HITE 

U.S.  GENERAL  ACCOUNTING  OFFICE 

FROM:  Acting  Director 

Office  of  Planning 

SUBJECT:  Draft  Audit  Report  on  the  United  States  Customs 

Service’s  Third  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 

Thank  you  for  providing  us  with  a  copy  of  your  draft  report  entitled 
“Customs  Service  Modernization:  Third  Expenditure  Plan  Meets 
Legislative  Conditions,  but  Cost  Estimating  Improvements  Needed”  and 
the  opportunity  to  discuss  the  issues  in  this  report. 

We  are  pleased  that  the  General  Accounting  Office  (GAO)  recognizes  the 
progress  we  are  making  to  satisfy  the  recommendations  from  their 
previous  reviews.  We  will  continue  to  address  all  open  recommendations 
aggressively. 

GAO  gives  particular  attention  to  the  cost  estimation  methodology 
supporting  the  third  expenditure  plan,  noting  that  we  are  still  in  the 
process  of  addressing  their  prior  recommendation  on  establishing  an 
effective  cost  estimating  program.  We  continue  to  pursue  efforts  to 
improve  our  cost  estimates  supporting  expenditure  plans.  Our  fourth 
expenditure  plan  will  be  supported  by  a  detailed  internal  review  of  the 
eCP  inputs  and  independent  expenditure  plan  cost  estimates.  Each  of 
these  efforts  will  be  based  on  the  Systems  Engineering  Institute’s  (SEI) 
guidance  for  validating  the  reliability  of  an  estimate.  We  are  confident 
that  these  efforts  will  provide  our  stakeholders  with  greater  confidence  in 
the  cost  estimates  supporting  expenditure  plan  4. 

Further,  Customs  is  proceeding  with  acquiring  the  services  of  an 
tradition  independent  contractor  to  develop  and  maintain  a  Modernization  Life 
Cycle  Cost  Model  which  will  then  support  all  planning  activities.  This 
*  model  will  also  be  based  on  the  SEI  estimating  guidance. 

S  ERVICE 


★ 


Page  47 


GAO-02-908  Customs  Service  Modernization 


Appendix  II 

Comments  from  the  U.S.  Customs  Service 


(310242) 


Page  48 


GAO-02-908  Customs  Service  Modernization 


GAO’s  Mission  The  General  Accounting  Office,  the  investigative  arm  of  Congress,  exists  to 

support  Congress  in  meeting  its  constitutional  responsibilities  and  to  help  improve 
the  performance  and  accountability  of  the  federal  government  for  the  American 
people.  GAO  examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of  accountability, 
integrity,  and  reliability. 


Obtaining  Copies  of 
GAO  Reports  and 
Testimony 


The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  the  Internet.  GAO’s  Web  site  (www.gao.gov)  contains  abstracts  and  full- 
text  files  of  current  reports  and  testimony  and  an  expanding  archive  of  older 
products.  The  Web  site  features  a  search  engine  to  help  you  locate  documents 
using  key  words  and  phrases.  You  can  print  these  documents  in  their  entirety, 
including  charts  and  other  graphics. 

Each  day  GAO  issues  a  list  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  list,  known  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  full-text  document  files.  To  have  GAO  e-mail  this 
list  to  you  every  afternoon,  go  to  www.gao.gov  and  select  “Subscribe  to  daily 
E-mail  alert  for  newly  released  products”  under  the  GAO  Reports  heading. 


Order  by  Mail  or  Phone  The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each.  A  check 

or  money  order  should  be  made  out  to  the  Superintendent  of  Documents.  GAO 
also  accepts  VISA  and  Mastercard.  Orders  for  100  or  more  copies  mailed  to  a  single 
address  are  discounted  25  percent.  Orders  should  be  sent  to: 

U.S.  General  Accounting  Office 
441  G  Street  NW,  Room  LM 
Washington,  D.C.  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 
TDD:  (202)  512-2537 
Fax:  (202)  512-6061 


To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 


Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 
E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 


Public  Affairs  Jeff  Nelligan,  managing  director,  NelliganJ@gao.gov  (202)  512-4800 

U.S.  General  Accounting  Office,  441  G  Street  NW,  Room  7149 
Washington,  D.C.  20548 
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